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Abstract 

We consider a Kullback-Leibler-based algorithm for the stochastic multi-armed bandit prob- 
lem in the case of distributions with finite supports (not necessarily known beforehand), 
whose asymptotic regret matches the lower bound of Burnetas and Katehakis (1996). Our 
contribution is to provide a finite-time analysis of this algorithm; we get bounds whose main 
terms are smaller than the ones of previously known algorithms with finite-time analyses 
(like UCB-type algorithms). 



1 Introduction 

The stochastic multi-armed bandit problem, introduced by Robbins (1952), formalizes the prob- 
lem of decision-making under uncertainty, and illustrates the fundamental tradeoff that appears 
between exploration, i.e., making decisions in order to improve the knowledge of the environment, 
and exploitation, i.e., maximizing the payoff. 

Setting. In this paper, we consider a multi-armed bandit problem with finitely many arms 
indexed by A, for which each arm a G A is associated with an unknown and fixed probability 
distribution v a over [0,1]. The game is sequential and goes as follows: at each round t ^ 1, the 
player first picks an arm A t € A and then receives a stochastic payoff Y t drawn at random according 
to VA t ■ He only gets to see the payoff Y t . 

For each arm a G A, we denote by /j, a the expectation of its associated distribution v a and we 
let a* be any optimal arm, i.e., a* G argmax fj, a . 

We write fi* as a short-hand notation for the largest expectation fi a * and denote the gap of the 
expected payoff fi a of an arm a G A to /z* as A a = /i* — /i a . In addition, the number of times each 
arm a G A is pulled between the rounds 1 and T is referred to as A^a), 

A T (a) d = f f> At=a} . 
t=l 

The quality of a strategy will be evaluated through the standard notion of expected regret, which 
we recall now. The expected regret (or simply regret) at round T ^ 1 is defined as 



R def 



= E 



l L A t 



= £)A a E[JV T (a)], 



(1) 



a£A 



where we used the tower rule for the first equality. Note that the expectation is with respect to the 
random draws of the Y t according to the va ± and also to the possible auxiliary randomizations that 
the decision-making strategy is resorting to. 

The regret measures the cumulative loss resulting from pulling sub-optimal arms, and thus quan- 
tifies the amount of exploration required by an algorithm in order to find a best arm, since, as (1) 
indicates, the regret scales with the expected number of pulls of sub-optimal arms. Since the for- 
mulation of the problem by Robbins (1952) the regret has been a popular criterion for assessing the 
quality of a strategy. 
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Known lower bounds. Lai and Robbins (1985) showed that for some (one-dimensional) para- 
metric classes of distributions, any consistent strategy (i.e., any strategy not pulling sub-optimal 
arms more than in a polynomial number of rounds) will despite all asymptotically pull in expecta- 
tion any sub-optimal arm a at least 

E ^( ffi »K^ + o(1) ) log(T) 

times, where K,{v a , v*) is the Kullback-Leibler (KL) divergence between v a and v*\ it measures how 
close distributions v a and v* are from a theoretical information perspective. 

Later, Burnetas and Katehakis (1996) extended this result to some classes of multi-dimensional 
parametric distributions and proved the following generic lower bound: for a given family V of 
possible distributions over the arms, 

E[N T (a)] > (-— + o(l)) log(T) , where /C inf (i/„, /x*) d = inf K{v a ,v) , 

with the notation E(v) for the expectation of a distribution v. The intuition behind this improvement 
is to be related to the goal that we want to achieve in bandit problems; it is not detecting whether 
a distribution is optimal or not (for this goal, the relevant quantity would be lC{v ai f*)), but rather 
achieving the optimal rate of reward /z* (i.e., one needs to measure how close v a is to any distribution 
v G V whose expectation is at least /x*). 

Known upper bounds. Lai and Robbins (1985) provided an algorithm based on the KL 
divergence, which has been extended by Burnetas and Katehakis (1996) to an algorithm based on 
/C; n f; it is asymptotically optimal since the number of pulls of any sub-optimal arm a satisfies 

E i* w i<Mb) +o|1| ) loe,r) - 

This result holds for finite-dimensional parametric distributions under some assumptions, e.g., the 
distributions having a finite and known support or belonging to a set of Gaussian distributions with 
known variance. Recently Honda and Takemura (2010a) extended this asymptotic result to the case 
of distributions V with support in [0, 1] and such that fi* < 1; the key ingredient in this case is that 

/C in f ("a,/**) is equal to /C min (^ a , fi*) = f in£ ve v-.E( v )^n* fc(v a , v). 

Motivation. All the results mentioned above provide asymptotic bounds only. However, any 
algorithm is only used for a finite number of rounds and it is thus essential to provide a finite- 
time analysis of its performance. Auer et al. (2002) initiated this work by providing an algorithm 
(UCB1) based on a Chernoff-Hocffding bound; it pulls any sub-optimal arm, till any time T, at 
most (8/A 2 ) logT + 1 + 7r 2 /3 times, in expectation. Although this yields a logarithmic regret, the 
multiplicative constant depends on the gap A 2 = (/i* — /j, a ) 2 but not on /Ci n f (v a , fi*), which can 
be seen to be larger than A^/2 by Pinsker's inequality; that is, this non-asymptotic bound does 
not have the right dependence in the distributions. (How much is gained of course depends on 
the specific families of distributions at hand.) Audibert et al. (2009) provided an algorithm (UCB- 
V) that takes into account the empirical variance of the arms and exhibited a strategy such that 
E[iVj'(a)] ^ 10(er 2 /A 2 + 2/A a ) logT for any time T (where u 2 a is the variance of arm a) ; it improves 
over UCB1 in case of arms with small variance. Other variants include the MOSS algorithm by 
Audibert and Bubeck (2010) and Improved UCB by Auer and Ortner (2010). 

However, all these algorithms only rely on one moment (for UCB1) or two moments (for UCB- 
V) of the empirical distributions of the obtained rewards; they do not fully exploit the empirical 
distributions. As a consequence, the resulting bounds are expressed in terms of the means \x a and 
variances a\ of the sub-optimal arms and not in terms of the quantity /Ci n f (v a , fi*) appearing in the 
lower bounds. The numerical experiments reported in Filippi (2010) confirm that these algorithms 
are less efficient than those based on /Ci n f . 

Our contribution. In this paper we analyze a /Ci n f-based algorithm inspired by the ones 
studied in Lai and Robbins (1985), Burnetas and Katehakis (1996), Filippi (2010); it indeed takes 
into account the full empirical distribution of the observed rewards. The analysis is performed 
(with explicit bounds) in the case of Bernoulli distributions over the arms. Less explicit but finite- 
time bounds are obtained in the case of finitely supported distributions (whose supports do not 
need to be known in advance). Finally, we pave the way for handling the case of general finite- 
dimensional parametric distributions. These results improve on the ones by Burnetas and Katehakis 
(1996), Honda and Takemura (2010a) since finite-time bounds (implying their asymptotic results) 
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are obtained; and on Auer et al. (2002), Audibert et al. (2009) as the dependency of the main term 
scales with /Cj n f (u a , /ji*). The proposed lC ln [ -based algorithm is also more natural and more appealing 
than the one presented in Honda and Takemura (2010a). 

Recent related works. Since our initial submission of the present paper, we got aware of two 
papers that tackle problems similar to ours. First, a revised version of Honda and Takemura (2010b, 
personal communication) obtains finite-time regret bounds (with prohibitively large constants) for a 
randomized (less natural) strategy in the case of distributions with finite supports (also not known 
in advance). Second, another paper at this conference (Garivier and Cappe, 2011) also deals with 
the /C-strategy which we study in Theorem 3; they however do not obtain second-order terms in 
closed forms as we do and later extend their strategy to exponential families of distributions (while 
we extend our strategy to the case of distributions with finite supports). On the other hand, they 
show how the /C-strategy can be extended in a straightforward manner to guarantee bounds with 
respect to the family of all bounded distributions on a known interval; these bounds are suboptimal 
but improve on the ones of UCB-type algorithms. 

2 Definitions and tools 

Let A" be a Polish space; in the next sections, we will consider X — {0,1} or X = [0,1]. We denote 
by V(X) the set of probability distributions over X and equip V(X) with the distance d induced by 
the norm |j • || defined by \\v\\ = snpj eC \ j x f dv\, where £ is the set of Lipschitz functions over X 1 
taking values in [—1, 1] and with Lipschitz constant smaller than 1. 

Kullback-Leibler divergence: For two elements v 1 k £ P{X), we write v <C k when v is abso- 
lutely continuous with respect to k and denote in this case by dv/dn the density of v with respect 
to k. We recall that the Kullback-Leibler divergence between v and n is defined as 

K,{v, k) = [ — log — dn if v <C k; and JC(u, k) = +oo otherwise. (2) 
J [01] dn dn 

Empirical distribution: We consider a sequence X\, X2, ... of random variables taking values in 
X, independent and identically distributed according to a distribution v. For all integers t ^ 1, we 
denote the empirical distribution corresponding to the first t elements of the sequence by 

1 * 

vt = - ■ 

1 8=1 

Non-asymptotic Sanov's Lemma: The following lemma follows from a straightforward adapta- 
tion of Dinwoodie (1992, Theorem 2.1 and comments on page 372). Details of the proof are provided 
in the appendix. 

Lemma 1 Let C be an open convex subset ofV(X) such that A(C) = inf JC{n, v) < 00 . 
Then, for all t ^ 1, one has W v {pt £ C} < e ~ tA ( c ) where C is the closure of C. 

This lemma should be thought of as a deviation inequality. The empirical distribution converges 
(in distribution) to v. Now, if (and only if) v is not in the closure of C, then A(C) > and the 
lemma indicates how unlikely it is that v t is in this set C not containing the limit v. The probability 
of interest decreases at a geometric rate, which depends on A(C). 

3 Finite-time analysis for Bernoulli distributions 

In this section, we start with the case of Bernoulli distributions. Although this case is a special case 
of the general results of Section 4, we provide here a complete and self-contained analysis of this 
case, where, in addition, we are able to provide closed forms for all the terms in the regret bound. 
Note however that the resulting bound is slightly worse than what could be derived from the general 
case (for which more sophisticated tools are used). This result is mainly provided as a warm-up. 

3.1 Reminder of some useful results for Bernoulli distributions 

We denote by B the subset of "P([0, 1]) formed by the Bernoulli distributions; it corresponds to 
B = ^({0, 1}). A generic clement of B will be denoted by /3(p), where p £ [0, 1] is the probability 
mass put on 1. We consider a sequence X\, X2, ■ . ■ of independent and identically distributed random 
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Parameters: A non-decreasing function / : N — S 
Initialization: Pull each arm of A once 
For rounds t + 1, where t ^ |.4|, 

- compute for each arm a e A the quantity 



= max| 9 e [0,1] : N t (a) )c(/3(fi atNt(a) ) , /%)) < f(t) J , 



£a,JV t (a) - ^ Y s ; 



WllCI C / '' ( I . . \ ; I ( M 

Ma) 



^t: A s =a 

in case of a tie, pick an arm with largest value of 'fi a ,N t (a)\ 
pull any arm A t +i G argmax B^ t . 



Figure 1: The /C-strategy. 

variables, with common distribution /3{p); for the sake of clarity we will index, in this subsection 
only, all probabilities and expectations with p. 

1 ' 

For all integers t ^ 1, we denote by p t — - X t the empirical average of the first t elements 

1 s=l 

of the sequence. 

The lemma below follows from an adaptation of Garivier and Leonardi (2010, Proposition 2). 
The details of the adaptation (and simplification) can be found in the appendix. 



Lemma 2 For all p G [0, 1], all e > 1, and all t^l, 

P p ^Ujs/c(/3(p s ),/3(p)) ^ £ }) ^ 2e ^ £logt \ e ~ £ ■ 
In particular, for all random variables N t taking values in {1, . . . ,t} 7 

F p {N t lc(p(p Nt ), f3( P )) > 4 < 2e [elogt] e~ s . 



Another immediate fact about Bernoulli distributions is that for all p G (0,1), the mappings 
JC. tP : q G (0, 1) h-> JC(P(j>), P(q)) and K Pi . : q G [0, 1] i-> K,(/3(q), (3(p)) are continuous and take 
finite values. In particular, we have, for instance, that for all e > and p G (0, 1), the set 



{«G[0,1]: lC((3(p),f3(q)) <e} 



is a closed interval containing p. This property still holds when p G {0,1}, as in this case, the 
interval is reduced to {p}. 

3.2 Strategy and analysis 

We consider the so-called IC-strategy of Figure 1, which was already considered in the literature, see 
Burnetas and Katehakis (1996), Filippi (2010). The numerical computation of the quantities B^ t 
is straightforward (by convexity of /C in its second argument, by using iterative methods) and is 
detailed therein. 

Before proceeding, we denote by a\ — /i a (l — Ma) the variance of each arm a G A (and take the 
short-hand notation a*- 2 for the variance of an optimal arm). 

Theorem 3 When /i* G (0, 1), for all non- decreasing functions f : N — > M.+ such that /(l) ^ 1, 
the expected regret Rt of the strategy of Figure 1 is upper bounded by the infimum, as the (c a ) aG ^ 
describe (0, +oo), of the quantities 

£ A ° ( vr^vWh + 4e E r/w M e ~ m + * 2 A2 (1 + r o) 4 2 + 3) • 

For ^i* = 0, its regret is null. For yu* = 1, it satisfies Rt < 2(|.4| — l). 
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A possible choice for the function / is f(t) = log((ei) log 3 (et)), which is non decreasing, satisfies 
/(l) ^ 1, and is such that the second term in the sum above is bounded (by a basic result about 
so-called Bertrand's series). Now, as the constants c a in the bound are parameters of the analysis 
(and not of the strategy), they can be optimized. For instance, with the choice of f(t) mentioned 
above, taking each c a proportional to (logT)" 1 / 3 (up to a multiplicative constant that depends on 
the distributions v a ) entails the regret bound 

V A log? 1 

a ^ Aa /C(/3( Ma ),/3(^)) +£T ' 

where it is easy to give an explicit and closed- form expression of e<r; in this conference version, we 
only indicate that Et is of order of (logT) 2 / 3 but we do not know whether the order of magnitude 
of this second-order term is optimal. 

Proof: We first deal with the case where /i* {0, 1} and introduce an additional notation. In view 
of the remark at the end of Section 3.1, for all arms a and rounds t, we let B~ t be the element in 
[0,1] such that 



KN t (a)),f3(q)) </(*)}= [B~ t , B+] 



q E [0,1] : N t (a) K (0(fia,N t (a)), /8(g)) < /(*) - fc, B+ ] . (3) 



As (1) indicates, it suffices to bound Nt(o) for all suboptimal arms a, i.e., for all arms such that 
[i a < We will assume in addition that fi a > (and we also have /x ^ fi* < 1); the case where 
Ha = will be handled separately. 

Step 1: A decomposition of the events of interest. For t > |.4|, when A t +i — a, we have 
in particular, by definition of the strategy, that B^ t ^ t . On the event 

{^+i = «} n {// g [s-* it) B+ tt ]} n [tia e [B~ t , B+ t ]} , 

we therefore have, on the one hand, h* ^ &t*,t ^ ^a,t an d on ^ ne other hand, B~ t ^ [ia ^ M*> that 
is, the considered event is included in j/x* G [-B^ t , B^ t ]|. We thus proved that 

{At+i = a} C g [B- v , S+ >t ] } U { Ma ^ [B~ t , B+ t ] } U { M * G [B" t , S+ ] } . 
Going back to the definition (3), we get in particular the inclusion 

{A t+1 = a} C |jV t (a*) /c(^(/x„*,iv t (a*)), /3(/x*)) > /(t)| 
U |iV t (o) /c(^(/x 0;JVt(0 )), /3(/i„)) > /(t) 

U ^A t (a) /c(/3(Ma,Ar t (a)), < /(*)} n {A t+1 = a}j . 

Step 2: Bounding the probabilities of two elements of the decomposition. We consider 
the filtration (Tt), where for all t > 1, the cr-algebra T t is generated by Ai,Y\,..., A t ,Y t . In 
particular, A t +i and thus all N t +i(a) are Tt -measurable. We denote by r 0) i the deterministic round 
at which a was pulled for the first time and by r 0i 2, t 0i 3, . . . the rounds t > |.4| + 1 at which a was 
then played; since for all fc > 2, 

T a , fc = min{t > |^| + 1 : iV t (a) = fc} , 

we see that {r a ^ = t} is Jt_i -measurable. Therefore, for each fc > 1, the random variable r aj fe is a 
(predictable) stopping time. Hence, by a well-known fact in probability theory (see, e.g., Chow and 
Teicher 1988, Section 5.3), the random variables X a ^ = Y Ta k , where fc = 1,2, . . . are independent 
and identically distributed according to v a . Since on {N t (a) = fc}, we have the rewriting 

1 k ~ 

fia,N t (a) = Va,k where Ha.k = £ Xa -3 ' 
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and since for t ^ \A\ + 1, one has N t (a) ^ 1 with probability 1, we can apply the second statement 
in Lemma 2 and get, for all t ^ \A\ + 1, 

pj JVt(a) /c(/3(/i a ,jv t (a)), /3(Ma)) > /(*)} < 2e \f(t)logt\ e~M . 

A similar argument shows that for all t ^ „4| + 1, 

N t (a*) /c(/3(/z * iJVt(a *)), /?(//)) > /(t)| < 2e [/(t) log t] e -'W . 

Step 3: Rewriting the remaining terms. We therefore proved that 

T-l T-l / , . 

E[JV T (a)] <l + 4eJ2 \f(t)logt\e- m + £ p| N t (a) /c(/3(M aiJVt(a) ) , /3(//)) < /(t) I n = a} 

and deal now with the last sum. Since / is non decreasing, it is bounded by 

T-l , s 

E n i A <+i = 4) where ^* = ^t(o) /c(/3(/z 0)JVt ( a )), /9(a**)) < f(T) . 



t=\A\ 



T-l 



Now, E P { Kt n = a l) 



E 



t=\A\ 



T-l 



E I {^ 1= .} 1 ^ 



t=|-4 



E 



k>2 



We note that, since N Ta k -i(a) = k — 1, we have that 



*r a . fc -l = {(fc - 1) /c(/3(Ma, fe -i), < /(T)} • 

All in all, since T a ,k ^ implies k ^ T — \A\ + 1 (as each arm is played at least once during the first 
| .A | rounds), we have 









Y-|.4|+l 


T-|^|+l , 


E 


k^2 


< E 


E 


= E p{ 






fc=2 


fc=2 



= £ P (*-l) K( j 8(ft,, fc _i) ) /3(A**)) </(T)L (4) 



Step 4: Bounding the probabilities of the latter sum via Sanov's lemma. For each 
7 > 0, we define the convex open set C 7 = j /?(<?) G £> : JC((3(q), (3(fi*)) < 7 j, which is a non-empty 
set (since fi* < 1); by continuity of the mapping IC. iM * defined after the statement of Lemma 2 when 

G (0, 1), its closure equals C 7 = {/%) G S : £(/?(?), ^ 7} . 

In addition, since [i a G (0, 1), we have that /C(/3(g), (3(fi a )) < oo for all q G [0, 1]. In particular, 
for all 7 > 0, the condition A(C 7 ) < oo of Lemma 1 is satisfied. Denoting this value by 



0„( 7 ) = inf |/C(/3(g), /3( M a)) : /8(g) G B such that £(/?(<?), /3(//)) < 7 \, 
we get by the indicated lemma that for all k ^ 1, 

pj/C(>(Ma,fe), /?(/«*)) < 7} = p{/?(M G C 7 } ^ e- ke ^ . 



Now, since (an open neighborhood of ) (3(fi a ) is not included in C 7 as soon as < 7 < /C(/3(/i a ), /3(/z*)), 
we have that a (-f) > for such values of 7. To apply the obtained inequality to the last sum in (4), 

(l + c a )/(T) 

K(j8(Ma), £(/**)) 

so that f(T)/k < /C(/3(Ai a ), /8(/i*))/(l + c a ) < £(/%„), for fc 5? fc , hence, 



we fix a constant c a > and denote by &o the following upper integer part, k 



E P{(fc - 1) /c(/3(Ma, fc -i), /?(aO) < /(T)} < E pj^Ovfc), /*(/**)) ^ — f 

fe=2 ^ fe=l ' 

T 

^ fco-l+E exp(-fc0 o (/(T)/fc)) . 



k—ko 
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Since 9 a is a non-increasing function, 

T T 

exp(-fc0 o (/(T)/fc)) < Yl exp(-fc0 a (/C(/3(xxa), /3(m*))/(1 + c a ) 



k—ko k—ko 



< r a ( Ca ) cxp(-fc o 0„(x;(/3(// a ), /3( M *))/(i + c a ))) <r a ( Ca ), 



where r a (c a ) - [l - exp(-0 a (/C(/3(/x a ), /8(/i*))/(l + Ca))) 
Putting all pieces together, we thus proved so far that 

E ^ & 1 + w^mk + ie %, [/(t) log<1 + ™ 

t—\A\ 

and it only remains to deal with T a (c a ). 

Step 5: Getting an upper bound in closed form for T a (c a ). We will make repeated uses 
of Pinsker's inequality: for p,q £ [0, 1], one has /C(/3(p), /3(g)) ^ 2 (p — q) 2 . 

In what follows, we use the short-hand notation a = a (/C(/3(/i a ), /3(/^,*))/(l + c a )) and there- 
fore need to upper bound 1/(1 — e _e>a ). Since for all u ^ 0, one has e~" ^ 1 — u + u 2 /2, we 

12 1 
get r o (c a ) ^ 7—7- — t — for 9 a < 1, and r a (c a ) < —r ^ 2 for 6 a ^ 1. It thus only 

B a (l - ©a/2) ©a 1 - e 1 

remains to lower bound a in the case when it is smaller than 1. 

By the continuity properties of the Kullback-Leibler divergence, the infimum in the definition of 
9 a is always achieved; we therefore let Jl be an element in [0, 1] such that 

9 a = /C(/3(m), fl/xa)) and JC (£(/!), /3(/x*)) = . 

it is easy to see that we have the ordering /x a < ju < /x*. By Pinsker's inequality, a ^ 2(/x — /x a ) 2 
and we now lower bound the latter quantity. We use the short-hand notation f(j>) = /C(/3(p), /3(/x*)) 
and note that the thus defined mapping / is convex and differentiable on (0, 1); its derivative equals 
f'(p) = l°g((l~ M*)/(M*))+l°g(xV(l — P)) forallp £ (0,1) and is therefore non positive for p ^ /x*. By 
the indicated convexity of/, using a sub-gradient inequality, we get /(/x) —f(fi a ) ^ /'(Ma) (/x — Ma) , 
which entails, since /'(Ma) < 0, 

~ /(g) - /(Ma) _ C a /(/l a ) 

M Ma 777 \ — i I 777 7 ' \°) 

f'(Ha) 1+Ca -/'(Ma) 

where the equality follows from the fact that by definition of /x, we have /(/x) = /(/x a )/(l + c a ). 
Now, since /' is differentiable as well on (0, 1) and takes the value at /x*, a Taylor's equality entails 
that there exists a £ G (xt a ,xt*) such that 

-/'(Ma) - / V) - /'(Ma) = /"(0 (m* - Ma) where /"(£) = l/£ + 1/(1 - £) = l/(£(l - 0) • 
Therefore, by convexity of r i-> r(l — r), we get that 

1 min{/x a (l - Ma), M*(l - M*)} 



-/'(Ma) M* - Ma 

Substituting this into (5) and using again Pinsker's inequality to lower bound /(/x ), we have proved 

M - n a > 2 -jj (/x* - /i a ) min{xta(l - /i a ), M*(l - M*)} ■ 
Putting all pieces together, we thus proved that 



(i + c y 

r a (c a ) 2 max <( — i '- — , 1 



> ; 



! c q (m* - Ma) 2 (min{/x a (l - /x a ), M*(l - M*)}) 
bounding the maximum of the two quantities by their sum concludes the main part of the proof. 
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Step 6: For ^* G {0, 1} and/or /j, a = 0. When /j,* = 1, then fi a *^N t (a*) = 1 f° r au i ^ + 1, 
so that Ba*. t = 1 for all t ^ + 1- Thus, the arm a is played after round t ^ |.4| + 1 only if = 1 
and fla.N t (a) = 1 ( m view of the tie-breaking rule of the considered strategy). But this means that a 
is played as long as it gets payoffs equal to 1 and is stopped being played when it receives the payoff 
for the first time. Hence, in this case, we have that the sum of payoffs equals at least T — 2(\A\ — 1) 
and the regret R T = E[7>* - (Yi + . . . + Y t )\ is therefore bounded by 2(\A\ - f). 

When At* = 0, a Dirac mass over is associated with all arms and the regret of all strategies is 
equal to 0. 

We consider now the case fi* G (0, 1) and [i a = 0, for which the first three steps go through; only 
in the upper bound of step 4 we used the fact that [i a > 0. But in this case, we have a deterministic 
bound on (4). Indeed, since /C(/3(0), = -log/z*, we have k /C(/?(0), /3(^*)) sC /(T) if and only 

^ fc < ffl = /( y ) 

" -log M * K^{fi a ),0i^J,*)) , 
which improves on the general bound exhibited in step 4. ■ 

Remark 1 Note that Step 5 in the proof is specifically designed to provide an upper bound on 
T a (c a ) in the case of Bernoulli distributions. In the general case, getting such an explicit bound 
seems more involved. 

4 A finite-time analysis in the case of distributions with finite support 

Before stating and proving our main result, Theorem 9, we introduce the quantity /C; n f and list some 
of its properties. 

4.1 Some useful properties of /C; n f and its level sets 

We now introduce the key quantity in order to generalize the previous algorithm to handle the case 
of distributions with finite support. To that end, we introduce "Pf([0,1]), the subset of ^([0, 1]) 
that consists of distributions with finite support. 

Definition 4 For all distributions v G Vf([0, 1]) and /i G [0, 1), we define 

K lni {v^)=m{\K{v,v'): v' eV F ([0,l}) s.t. E(i/)>n}, 

where E(v') = Jj ^ xdv'{x) denotes the expectation of the distribution v' . 

We now remind some useful properties of /C; n f. Honda and Takemura (2010b, Lemma 6) can be 
reformulated in our context as follows. 

Lemma 5 For all v G ^([0,1]), the mapping JCi n f(v, •) is continuous and non decreasing in its 
argument [i G [0,1). Moreover, the mapping /Ci n f(-,/i) is lower semi- continuous on Vf([0, 1]) for 
all fie [0,1). 

The next two lemmas bound the variation of /Cj n f , respectively in its first and second arguments. 
(For clarity, we denote the expectations with respect to v by E„.) Their proofs are both deferred to 
the appendix. We denote by || • \ x the ^ 1 -norm on "P([0, 1]) and recall that the £ 1 -norm of v — v' 
corresponds to twice the distance in variation between v and v 1 . 

Lemma 6 For all /j, G (0, 1) and for all v, v' G Pf([0, 1]), the following holds true. 

- In the case when E„[(l — /i)/(l — X)~\ > 1, then /C; n f(^,/i) — /Ci n f (v 1 , At) M u ^ \\v — v'\\i , for 
some constant M u ^ > 0. 

- In the case when E„[(l — — X)] < 1, the fact that K.- m i{v, fi) — JC m f {v' ,n) ^ aK. m t{v, (i) 
for some a G (0,1) entails that 

(2/a) ((2/a) - l) ' 

Lemma 7 We have that for any v G ^([0,1]), provided that fi ^ \i — e > E{y), the following 
inequalities hold true: 

e/(l - n) > /C inf {v, t 1 ) - tCini{v, A* - e) > 2e 2 
Moreover, the first inequality is also valid when E{v) ^ [i > fi — e or fj, > E{v) > \i — e. 
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Parameters: A non-decreasing function / : N — > R 
Initialization: Pull each arm of A once 
For rounds t + 1, where t ^ |.4|, 

- compute for each arm a 6 A the quantity 

Ba, t = max jg 6 [0, 1] : iV t (a) /C inf (J? 0iJVt(o) , (?) < /(i)} , 

where ^o,Jv t (o) = XI 5y * ; 

- in case of a tie, pick an arm with largest value of /Lt a ,jv t (a)i 

- pull any arm A t +i G argmax B^ t . 

Figure 2: The strategy IC- m f . 

Level sets of JC m f. For each 7 > and [i G (0, 1), wc consider the set 
C Ml7 = \y' 6P F ([0,1]) : AC inf (i/',At) < 7} 

= {i/eP f ([0,l]): 3^^(10,1]) s.t. £7(^)>A* and /C(z/', ^) < 7} • 
We detail a property in the following lemma, whose proof is also deferred to the appendix. 
Lemma 8 For all 7 > and [i G (0, 1), the set is a non-empty open convex set. Moreover, 

C M , 7 D {1/ G TV ([0,1]) : /C inf (z/,/x) 7} • 
4.2 The /Ci n f —strategy and a general performance guarantee 

For each arm aeA and round t with N t (a) > 0, we denote by ^ a ,N t (a) the empirical distribution of 
the payoffs obtained till round t when picking arm a, that is, 



where for all x G [0, 1], we denote by S x the Dirac mass on x. We define the corresponding empirical 
averages as 

V-a*M t (a*) = £(«V,iV t (a*)) = J^TT ^ " 

We then consider the /Ci n f -strategy defined in Figure 2. Note that the use of maxima in the definitions 
of the B^t i s justified by Lemma 5. 

As explained in Honda and Takemura (2010b), the computation of the quantities /C; n f can be 
done efficiently in this case, i.e., when we consider only distributions with finite supports. This 
is because in the computation of /C; n f, it is sufficient to consider only distributions with the same 
support as the empirical distributions (up to one point). Note that the knowledge of the support of 
the distributions associated with the arms is not required. 

Theorem 9 Assume that u* is finitely supported, with expectation fi* G (0, 1) and with support 
denoted by S* . Let a G A be a suboptimal arm such that \i a > and v a is finitely supported. Then, 
for all c a > and all 

< s < min (a o , -^-(1 - M*) Kim K, M*)} , 
the expected number of times the /Ci n f -strategy 7 run with f(t) — logt, pulls arm a satisfies 
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where 



' log T e 
® a (c a ,e) = 9 a ( —j— + - - — ) with k 



(l + c n ) logT 

/C in f(l> ,//*) 



and /or a// 7 > 0, 

e o (7) = inf{/C(i/,z/ a ) : 1/ s.t. /C inf (z/, //) < 7} . 
As a corollary, we get (by taking some common value for all c a ) that for all c > 0, 

R T ^ > A a — — — + h{c) , 

t^A ^K>M*) 

where h(c) < 00 is a function of c (and of the distributions associated with the arms), which is 
however independent of T. As a consequence, we recover the asymptotic results of Burnetas and 
Katehakis (1996), Honda and Takemura (2010a), i.e., the guarantee that 

R T . A a 
lim sup ^ > — — . 

t^oo logT ^ lC int {u a ,fj,*) 

Of course, a sharper optimization can be performed by carefully choosing the constants c a , 
that are parameters of the analysis; similarly to the comments after the statement of Theorem 3, we 
would then get a dominant term with a constant factor 1 instead of 1 + c as above, plus an additional 
second-order term. Details are left to a journal version of this paper. 

Proof: By arguments similar to the ones used in the first step of the proof of Theorem 3, we have 

{A t+ l C {[1* -S < fi a ,N t (a)} U {/i*-e>B+* jt } u{/-££ [Pa,N t (a), B+,t]}'> 

indeed, on the event {A+i = a} H j/i* — e ^ M a ,JV t (a)| ^ |m* — e ^ ^a*.tj i 

we have, fi a .N t (a) ^ M* ~ e ^ ^o*,t ^ ^<£t (where the last inequality is by definition of the strategy). 
Before proceeding, we note that 

{V - e E [fia,N t (a), Bit] } C jiVt(a) /C inf (9 ajA r t(a) , (J-* — e) ^ /(*)} , 

since /Ci„f is a non-decreasing function in its second argument and /C; n f (y, E{v)~) = for all distri- 
butions v. Therefore, 

T-l T-l 



E[N T {a)] < 1 + Y, P {m* - £ < A*a,jv t (a) and A t+1 - a} + ^ p{// - e > B+ >t } 

t=|^| t=\A\ 
T-l 

+ F {^t( a ) £inf(£a,JV t (a)> - e) < /(*) and A t+1 = a} 



*=l-4| 

now, the two sums with the events "and A t +i — a" can be rewritten by using the stopping times r a .fe 
introduced in the proof of Theorem 3; more precisely, by mimicking the transformations performed 
in its step 3, we get the simpler bound 

T-l-AI + l T-l 

E[N T (a)] sc 1+ ]T p{ M *-£<Mca-i}+ F {^- £>B a*,t} 

k=2 " t=\A\ 

T-\A\ + 1 



]T f{(k-l)K iBl (u a , k - 1 ,n*-e)^f(t)}, (6) 



k=2 



where the v a , s and fi a , s are respectively the empirical distributions and empirical expectations com- 
puted on the first s elements of the sequence of the random variables X a j = Y Ta , which are i.i.d. 
according to v a . 
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Step 1: The first sum in (6) is bounded by resorting to Hocffding's inequality, whose appli- 
cation is legitimate since /i* — jj, a — e > 0; 



r— |-4|+i 



T-\A\ 



E P {M* - £ < Ma,fc-l} = E F { fl * ~ £ < Va,k -Ma} 

k=l 
T-\A\ 

< V p -2fe(M*-Ma-e) 2 < f < L 



-e 



where we used for the last inequality the general upper bounds provided at the beginning of step 5 
in the proof of Theorem 3. 

Step 2: The second sum in (6) is bounded by first using the definition of B^* t , then, 
decomposing the event depending on the values taken by N t (a*); and finally using the fact that on 
{ N t (a* ) = Ac}, we have the rewriting v a ,N t (a) = v a .k and ^a,N t {a) = Va,k ; more precisely, 



T-l 



T-l 



E p{m* - s > B+ t ) ^ Yl P {^K) Kintfc'W), V*-s)> f(t)} 

t=\A\ ' t=\A\ 

T-l t 

= E E P {^( a *) = fc and k *> ^ - e) > /(*)} 

t=|^| k=i 

T T-l 

< E E P { fc Km* («V,k, A** - e) > /(*)} • 
fc=lt=I.AI 

Since / = log is increasing, we can rewrite the bound, using a Fubini-Tonelli argument, as 

T-l T T-l , ^ 

E P{/i*-e>B+ v } < E E P H _1 ( fc/Cinf (^. fe '^- £ )) >*| 
t=|-4| fe=l t=UI ^ 



k=l 



{/Cinf /J*-e)>o} 



Now, Honda and Takemura (2010a, Lemma 13) indicates that, since /i* — e G [0, 1), 

SUp /C inf (z/, - £) ^ l0g(l/(l — A** + £)) == ^max ; 
i/€Pp([0,1]) 

we define Q = K max /e 2 and introduce the following sets (V^i^^q: 

V q = {veV F ([0,i\) ■ (q-l)e 2 </C in t(v,fi* -e) ^qe 2 }. 
A peeling argument (and by using that / _1 = exp is increasing as well) entails, for all k ^ 1, 

f~ 1 (klC int (v a *, k , fi* -e 



E 



(7) 



{/C inf O a *, fc , /J*-£)>0} 

q r 

= E E / -1 (fc£inf(iv,fc,/i*-e)) 

g=l L g 
Q Q 

^ E P l^ e ^} / _1 (^£ 2 ) < E P { X; - f H*-e)>(q- l)e 2 } /^(fcge 2 ) , (8) 

g=l o=l 

where we used the definition of V q to obtain each of the two inequalities. Now, by Lemma 7, when 
E(v a *,k) < — e, which is satisfied whenever /Ci n f (p a *,k, A 4 * — £ ) > 0) we have 

A^inf {p a *,k, A 4 * — e ) ^ ^inf (^a*,fc, A 4 *) _ 2 £ 2 < JC{u a *,k, V*) ~ 2e 2 , 

where the last inequality is by mere definition of /Ci n f. Therefore, 

P{/C inf (iv ifc , m* - e) > (g - 1)£ 2 } «S P{/C(^, fc , i/*) > (g + l)e 2 } . 
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We note that for all jfe > 1, p{/C(£ a *, fe , v*) > (q + l)e 2 } < (k + l) 15 * 1 e^^ 1 ^ 2 , 

where we recall that S* denotes the finite support of v* and where we applied Corollary 12 of the 
appendix. Now, (8) then yields, via the choice / = log and thus / _1 = cxp, that 



E 



e -k(q+l)e 2 e kqe 2 



=Q(fe+l)l s *l e- k ^ 2 



Substituting the value of Q, we therefore have proved that 

T-l „ , \ T 



E ^ - - > < i iog( r -^— ) E( fc + 

t=i^i v ' 7 fe=i 

Step 3: The third sum in (6) is first upper bounded by Lemma 7, which states that 

£inf (y a ,k-\, A**) - - M*) < ^inf (v a ,k-i, M* - e) , 
for all k ^ 1, and by using /(i) < f(T); this gives 

T-|^| , . T-|^| 

^ P * /C inf (? a , fe , a**) < /(T) + T -^ T \ = e <V,7*} 

fe=i ^ ^ - 1 fe=i 



fc=i 



where 7^ = f(T)/k + e/(l — /x*) and where the set C M *, 7fc was defined in Section 4.1. For all 7 > 0, 
we then introduce 

#a (7) = inf{/C(i/',i/ a ) : 1/ eC M *, 7 j =inf j/C(i/',i/ a ) : i/'eC M * i7 |, 

(where the second equality follows from the lower semi-continuity of K.) and aim at bounding 

As shown in Section 4.1, the set C M *, 7 is a non-empty open convex set. If we prove that a (7) 
is finite for all 7 > 0, then all the conditions will be required to apply Lemma 1 and get the upper 
bound 

T-\A\ T-\A\ 

E *{7«ke*r.T>}< E e ~ fcM7fc) - 
fe=i fc=i 

To that end, we use the fact that ^ a is finitely supported. Now, either the probability of interest 
is null and we are done; or, it is not null, which implies that there exists a possible value of v a ^ that 
is in C M * >7 ; since this value is a distribution with a support included in the one of v a , it is absolutely 
continuous with respect to v a and hence, the Kullback-Leibler divergence between this value and v a 
is finite; in particular, O (7) is finite. 

-(l + Ca)f(TY 



/Cinf(fa,M*) 



Finally, we bound the 9 a ("fk) for values of k larger than ko 
we have that for all k ^ ko, in view of the bound put on e, 

f(T) , £ ^ AC inf (i/„At*) c a /2 1 + c a /2 

lk ^ lka = ~kV i + Ca + TT^ ^ " } = TT^7 ^ ( "°' " } • (9) 

Since 9 a is non increasing, we have 

T-\A\ t-\a\ 



fc=i 



k—ko 



provided that the quantity a (c a ,e) = a (7fe o ) is positive, which we prove now. 
Indeed for all v' G C AI * j7(eo , we have by definition and by (9) that 

£ inf (i/,/z*) -/C in f(i/ ,At*) < 7fe -X: in f(i/ ,At*) < - ((c /2)/(l + c ))/C in f(i/ ,At*) . 
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Now, in the case where E„ a [(1— JQ] > 1, we have, first by application of Pinsker's inequality 

and then by Lemma 6, that 



W -v, " 2 



C 2 a {iCinijVa, H*)f 



/C(l/,I/ a ) ^ > —J (/C inf (f a , //*) -/C inf (l/',/i*)) > 



since, again by Pinsker's inequality JC- m f (v a , A**) ^ (Ma — M*) 2 /2 > 0, we have exhibited a lower 
bound independent of v' in this case. In the case where E„ [(1 — /U*)/(l — -X')] < 1, we apply the 
second part of Lemma 6, with a a — (c a /2)/(l + c a ), and get 

Thus, in both cases we found a positive lower bound independent of i/', so that the infimum over 
v' E £V, 7fe of the quantities K.[ n f (v', fi*), which precisely equals 9 a ("fk a )i 1S also positive. This 
concludes the proof. ■ 

Conclusion. We provided a finite-time analysis of the (asymptotically optimal) /Qnf-strategy in the case 
of finitely supported distributions. One could think that the extension to the case of general distributions 
is straightforward. However this extension appears somewhat difficult (at least when using the current 
definition of /Cj n f) for the following reasons: (1) Step 2 in the proof uses the method of types, that would 
require some extension of Sanov's non- asymptotic Theorem to this case. (2) Step 3 requires to have both 
0a (7) < 00 for all 7 > and #0(7) > for 7 < /Ci n f (u a , (J-*), which does not seem to be always the case 
for general distributions. Exploring other directions for such extensions is left for future work; for instance, 
histogram-based approximations of general distributions could be considered. 
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A Appendix beyond the COLT page limit 

A conference version of this paper was published in the Proceedings of the Twenty- Fourth Annual 
Conference on Learning Theory (COLT'll); this appendix details some material which was alluded 
at in this conference version but could not be published therein because of the page limit. 

A.l Proof of Lemma 2 

We only provide it for the convenience of the readers since it is similar to the one presented in Garivier 
and Leonardi (2010, Proposition 2) or in Garivier and Cappe (2011); it was however somewhat sim- 
plified by noting that the proof technique used leads to a maximal inequality, as stated in Lemma 2, 
and not only to an inequality for a self-normalized average, as stated in the original reference. 

Proof: The result is straightforward in the cases p = or p = 1, since then, p s = p almost surely; 
in the rest of the proof, we therefore only consider the case where p G (0, 1). 

It suffices to show the first bound stated in the lemma, since the second one follows by a decom- 
position of the probability space according to the values of N t . Actually, we will show 

Pp^U{ S /C(>(p s ),/3(p)) and p s >pJ^J ^e[elogt]e- £ , 

and the desired result will follow by symmetry and a union bound. 

Step 1: A martingale. For all A > 0, we consider the log-Laplace transform 

%{X) = logE p [e AXl ] = log((l -p) +pe x ) , 

with which we define the martingale 

W s (\) = exp(A(X! + . . . + X s ) - s V P (A)) . 

Step 2: A peeling argument. We introduce to = 1 and t k = |_7 fc J; f° r some 7 > 1 that will 
be defined by the analysis. We also denote by K = [(log t)/ (log 7)] an upper bound on the number 
of elements in the peeling. 

We also note that by continuity of the Kullback-Leibler divergence in the case of Bernoulli 
distributions, for all e > 0, there exists a unique element p £ E (p, 1) such that A^(/3(g e ), /3(p)) = e; 
this element satisfies that 

/C(/3(g), /3(p)) ^ e and q ^ p entails q ^ p £ . 
Denoting by Ek — e/tk, a union bound using the described peeling then yields 



Pp^U ja/c(/3(p s ),/3(p)) > £ and P»>P^j 

(J [ s ^(Ps), (3(p)) > £ and p s > pj 




S=tk-1 

t 



|J \lC(p(p.),P(pj)>^ and Ps> P \ 

=tfc-l 

tfc \ K I t k 

= E p p( U {p^ptM -E p p U {xi+...+x s - S p £k >o} 

fc-l / fe=l \s=t k _! 

Now, the variational formula for Kullback-Leibler divergences shows that for all k, there exists a A^ 
such that 

e k = /C(/3(p £ J, (3(p)) = \ k p £k - ^p(A fe ) ; 
actually, a straightforward calculation shows that Afe = log(p £jc (l — p) — log(p(l — p £k )) > is a 
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suitable value. Thus, 

K 



J2 f p[ U {xi + ...+x s - sPek ^0} 

k=i ys=t fc _i 

X> P ( (J {exp(A fc (X 1 + ... + X s )-A fe sp £fe ) > l} 

fc=l \s— , 

E p p( U {exp(A fe (X 1 + ... + X s )-sV P (A fe )) ^e S£fc } 
fe=i ys=t fc _i 

< E p pf U {^( A *) >z tk - i£k ) 



k=l \s=tfc_i 
A' 



< J2 e' tk - l£k = Ke- £ l~< , 



fe=i 



where in the last step, we resorted to Doob's maximal inequality. 

Step 3: Choosing 7. The obtained bound equals, by substituting the value of K and by 
choosing 7 = e/(e — 1), 



Ke- £ l^ [(logt)/(log 7 )l e 



e+l _ 



log* 



log(e/(e-l)) 



the proof is concluded by noting that e > 1 1 — » log(e/(£ — 1)) — 1/e is decreasing (its derivative is 
negative), with limit at +00. ■ 

A. 2 Details of the adaptation leading to Lemma 1 

The exact statement of Dinwoodie (1992, Theorem 2.1 and comments on page 372) is the following. 

Lemma 10 (Non-asymptotic Sanov's lemma) Let C be an open convex subset of V(X) such 
that 

A(C) = inf K.(k, v) < 00 . 

Then, for all t ^ 1, 

We show how it entails Lemma 1. Let C be an open convex subset of V(X) and let C be its 
closure. We denote by 

C s = {veC: d{v,C)<5) 

the 5-open neighborhood of C, we have C C Cs for all 5 > 0. Therefore, by the lemma above, since 
A(C S ) ^ A(C) < 00, 

r v {v t eC}< P„{P t G C 5 } e-* A ( c ^ . 

We pick for each integer n > 1 an element /t„ such that A(Ci/„) = fC(K n ,v) — 1/n; by Dinwoodie 
(1992, proof of Proposition 1.1), the sequence of the K n admits a converging subsequence n v (n), 
whose limit point Koo belongs to C and which satisfies 

/C(koo, v) < liminf /C(/t n , v) = liminf A(Cs) ■ 

n— >oo 5— >0 

Therefore, by taking limits in the above inequality, we have proved the desired inequality, 

P„{V t eC}^ e -*K(«oo,^) ^ e -tA(c) _ 
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A. 3 Useful properties of K, m f and its level sets 

Proof of Lemma 6: We resort to the formulation of /C; n f in terms of a convex optimization 
problem as introduced in Honda and Takcmura (2010b); more precisely, it is shown therein that 

/C inf (v, n) = max |e„ [log(l + X(p - X))] : A E [0, 1/(1 - /*)] J (10) 

(where X denotes a random variable distributed according to v), as well as the following alternative. 
The optimal value A„ of the parameter A indexing the set is equal to 1/(1 — /i) if and only if 
E„ [(1 - - X)] ^ 1, and lies in [0, 1/(1 - fi)) if E„ [(1 - /i)/(l - X)] > 1. 

For all A G [0, 1/(1 — a*)] , we now introduce the function 

4>x : x 6 [0, 1] i— > log(l + X(p - x)) , 

which is always continuous on [0, 1); we note also that it is continuous and finite at x — 1 when 
A < 1/(1 — fi)- In the latter case, <j)\ is bounded; since it is decreasing, it is easy to get a uniform 
bound: for all x, 

\4>x{x)\ < 1 0(0) I + 1 0(1) I = log 1 + * + ^ M\ . 

It then follows that for all A e [0, 1/(1 - //)), 

E V [MX)] - K>[MX)] <M A ||i/-i/||i. (11) 

In the case when \ v < 1/(1 — /j,), we have from the variational formulation (10) that 

/CinfOA^-ZCinfO/,/!) <E„[0 Al/ (A-)] -E^[0 A „(X)] <M A „ 

Thus, the constant M„ i([J in the statement of the lemma corresponds to our quantity M\ v in this 
case. 

We now consider the case where \ v = 1/(1 — /i). By (11) and variational formulation (10), we 
have that for all A G [0, 1/(1 - fj,)), 

/C in f {V, /i) - /C in f (^', A*) < /Cinf (^ A*) ~ [<M-X")] 

At) - E„ [MX)] ) + (e„ [0 a (X)] - E„i [MX)] 

The second difference is bounded according to (11); the first difference is bounded by concavity of 
A < 1/(1 -a*) 4>\{x), for all x: 

E„ [0 A (X)] > (1 - A(l - /!)) E„ [0 o (Af )] + A(l - M ) E„ [0 O (*)] 

= A(l - n) E„ [0i/(i_ M ) (X)] = A(l - At) /Ci n f (z/, A*) , 

since </>o is the null function and \ v = 1/(1 — /j,). Putting all pieces together, we have proved that 
for all A e [0, 1/(1 - At)), 

/C in f(zAA*) - /C in f(^',At) < (l - A(l - At)) /Ci n f (i/, /i) + M A ||zv - . (12) 

We recall that by assumption, /C; n f(^, A 1 ) ~ /Cinf (^', A*) ^ a /Cinf (^, A 4 ) with a € (0,1), so that the 
choice A = (1 — a/2)/(l — /i), which indeed lies in (0, 1/(1 — At)), is such that 

M —4 + TTA^T))='°< 1 + ^)^' 

so that (12) entails 

a /Cinf {v, At) < 77 /Cinf (^, A*) + — Ik - 1/7 111 ) 
2 a 



and finally 



, a 2 a 2 (l — At) 1 — A* 

f — ^ 1 ^ TT ~ ~ 



4A l-a/2 (2/a) ((2/a) - l) 
which concludes the proof. 
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Proof of Lemma 7: In Honda and Takemura (2010b) it is shown that in this case, /Ci n f(f, fi) is 
diffcrcntiablc in \i e {E(v), 1) with 

1 d „ , , a — E(v) , s 

—^-K^^^Al, ( 13 ) 

We apply this result to the rewriting 

f 1 d 

K, in f(i/,fi)-lC in i(u,ij,-e)= / — JCi n f(v,u}du, 

J H-e "A* 

which already gives one part of the bound. For the lower bound, we note that by assumption 
—E{v) > — (fi — e) and that u(l — u) ^ 1/4 (since we consider distributions with support included 
in [0, 1]); so that, for all u, 

Integrating the bound concludes the main part of the proof. 

Now, to see that the first inequality in the statement is always valid, we need to consider the 
case when E(v) ^ /x, for which the statement is trivial since then /Ci n f(^, /i) = 0, and the case when 
yu > E(v) ^ \i — e. But in the latter case, it is shown in Honda and Takemura (2010b, Lemma 6, 
case 2) that 

/Cinf [U, /Li) < — — , 

1 /i 

which concludes the proof. ■ 

Proof of Lemma 8: First, C M (7) is non empty as it always contains 5^, the Dirac mass on \i. 

The fact that ^(7) is convex follows from the convexity of K, in the pair of probability distri- 
butions that it takes as an argument. Indeed, for all a e [0, 1], v', v" E C^(-f), denoting by v' , v'^ 
some distributions such that the defining conditions in (^(7) are satisfied, we have that 

E{av' ll + (1 - a)v£) > n 

and 

K{av> + (1 - a)u", + (1 - a)u^) < aK{y' , ^) + (1 - a)K{v", < 7 . 

We prove that (^(7) is an open set. With each v' e ^(7), we associate a distribution 1/ 
satisfying the defining constraints in ^(7); by choosing 

a = g (0, 1/2), 

we have that the open set formed by the 

(l-a)i/ + av", i/'gB(i/,1) 

is contained in C^, where B(z/, 1) denotes the ball with center v' and radius 1 in the norm || • || 
over V{X). Indeed, we have on the one hand, 

E((l - v„ + a v ) > (1 - a) £(zvj > I 1 I £(z/J = > /x , 

and on the other hand, by convexity of the Kullback-Leibler divergence, 

/C((l -a)i/ + ai/', (l-a)i/ +az/') ^ (l-a)K(i/, v'^) < (1 - a) 7 . 

To prove the desired inclusion, we first note that in the case of Vp(\0, 1]), Honda and Takemura 
(2010b) show that one has the rewriting 

/Ci n f (u, fi) = min |/C(^, v') : v 1 

GV F ([0,1]) s.t. £(z/)>/x}; 
in particular, the infimum is achieved with this new formulation. Hence, 

C„, 7 = {i/eP F ([0,l]): 3i/„e?V([0,l]) s.t. E^^n and /C^', ^) < 7} • 
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Also, an element of the set of interest is therefore a v' £ "Pf([0, 1]) such that fCi n f(u', p) ^ 7, that 
is, such that there exists v'^ £ V([0, 1]) with E{y'^) ^ f-i and /C(z/, i^) ^ 7. Now, the distributions 

1 I v' H — <5i , thanks to the v' = ( 1 ) i/„ H — <5i , 



n 



all belong to C 7 , as, similarly to the above argument, 

E «) > M+ ^ > M and £«, <„) < (l - I) JC(i/,i/„) < 7. 

In addition, we have by construction that the v' n converge to z/, hence, v' £ C 7 . ■ 
A. 4 The method of types 

Let Xi, A" 2 , . . . be a sequence of random variables that are i.i.d. according to a distribution denoted 
by v. In this subsection, we will index all probabilities and expectations by v. 

For all k ^, we denote by £k the set of possible values (the so-called types) of the empirical 
distribution 

k 

»k = Y; s * ■ 
.?=i 

If v has a finite support denoted by S, then the cardinality |£fc| of £k is bounded by (k + 1)I S L 
Lemma 11 In £/ie case where v has a finite support, for all k ^ 1 and n £ £k, 

K{v k = K } ^ e - kK ^ v K 
Corollary 12 In f/ie case where v has a finite support, for all k ^ 1, all j > 0, 

P{/C(p fc , 1/) > 7} = J! I {/C( K ^)> 7 } P ^{ 9 fe = «} 

< E e- fe ^) < |5 fc | e"^ (k + I)' 5 ' e"^ . 
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